Physical Representation Learning and Parameter Identification from Video Using Differentiable Physics

Abstract Representation learning for video is increasingly gaining attention in the field of computer vision. For instance, prediction models enable activity and scene forecasting or vision-based planning control. In this article, we investigate combination differentiable physics spatial transformers a deep action conditional representation network. By our model learns physically interpretable latent can identify physical parameters. We propose supervised self-supervised methods architecture. experiments, consider simulated scenarios with pushing, sliding colliding objects, which also analyze observability properties. demonstrate that network learn to encode images properties like mass friction from videos sequences. evaluate accuracy training methods, ability method predict future frames input actions.